Fields of Gold: Scraping Web Data for Marketing Insights

Marketing scholars increasingly use web scraping and application programming interfaces (APIs) to collect data from the internet. Yet, despite widespread of such data, idiosyncratic sometimes insidious challenges in its collection have received limited attention. How can researchers ensure that sets generated via APIs are valid? While existing resources emphasize technical details extracting authors propose a novel methodological framework focused on enhancing validity. In particular, highlights how addressing validity concerns requires joint consideration legal/ethical questions along three stages collecting data: selecting sources, designing collection, data. The further review more than 300 articles using published top five marketing journals offer typology advanced thought. article concludes with directions for future research identify promising sources embrace approaches capture describe evolving marketplace realities.